[3.2] Create a simple benchmarking framework and use it to benchmark crypto primitives #17

linh2931 · 2022-08-17T21:56:32Z

Resolve eosnetworkfoundation/mandel#786.

This PR is built upon eosnetworkfoundation/mandel#799. Changes to incorporate the comments from that PR (eosnetworkfoundation/mandel#799) are

Add benchmarking sha1, sha256, sha512, ripemd160, sha3-256, keccak256 hashing, k1/r1/webauthnkey signing/recovering, and blake2
Use std::chrono::high_resolution_clock
Add double quotes around ${CMAKE_CURRENT_SOURCE_DIR} in CMakeList.txt
Various minor output format and name changes

JSON format output will be done in a future improvement task.

Help

./benchmark --help
benchmark command line options:
  -f [ --feature ] arg      feature to be benchmarked; if this option is not
                            present, all features are benchmarked.
  -l [ --list ]             list of supported features
  -r [ --runs ] arg (=1000) the number of times running a function during
                            benchmarking
  -h [ --help ]             benchmark functions, and report average, minimum,
                            and maximum execution time in nanoseconds

Benchmarking all functions

./benchmark
function                    runs        average       minimum       maximum

alt_bn_128:
alt_bn128_add               1,000       5,270 ns       3,694 ns     129,168 ns
alt_bn128_mul               1,000     240,319 ns     237,491 ns     256,501 ns
alt_bn128_pair (1 pair)     1,000   6,401,073 ns   6,360,615 ns   6,548,664 ns
alt_bn128_pair (10 pairs)   1,000  25,339,325 ns  25,218,417 ns  25,668,534 ns

blake2:
blake2                      1,000         318 ns         305 ns       1,389 ns

hash:
sha1 (64 bytes)             1,000         141 ns         131 ns       6,139 ns
sha1 (4096 bytes)           1,000       2,848 ns       2,835 ns       6,540 ns
sha256 (64 bytes)           1,000         151 ns         147 ns         897 ns
sha256 (4096 bytes)         1,000       3,286 ns       3,278 ns       3,425 ns
sha512 (64 bytes)           1,000         323 ns         301 ns       4,534 ns
sha512 (4096 bytes)         1,000       7,307 ns       7,240 ns       8,967 ns
ripemd160 (64 bytes)        1,000         545 ns         532 ns       4,245 ns
ripemd160 (4096 bytes)      1,000      16,042 ns      15,986 ns      17,815 ns
sha3-256 (64 bytes)         1,000       2,515 ns       2,492 ns       3,276 ns
sha3-256 (4096 bytes)       1,000      93,361 ns      87,823 ns     103,785 ns
keccak256 (64 bytes)        1,000       2,514 ns       2,491 ns       2,627 ns
keccak256 (4096 bytes)      1,000      93,962 ns      91,291 ns      99,095 ns

key:
k1_sign_non_canonical       1,000      40,160 ns      39,806 ns      92,056 ns
k1_recover                  1,000      46,592 ns      45,789 ns     112,687 ns
r1_sign                     1,000     308,642 ns     226,944 ns     861,173 ns
r1_recover                  1,000     108,845 ns     107,913 ns     116,391 ns
webauthn_sign               1,000     290,216 ns     207,265 ns     385,395 ns
webauthn_recover            1,000     110,448 ns     109,612 ns     120,243 ns

modexp:
1024 bit width              1,000     389,245 ns     388,228 ns     402,387 ns
2048 bit width              1,000   2,914,965 ns   2,911,569 ns   2,932,296 ns

Benchmarking a single feature with a different number of repeats from the default 1,000

./benchmark -f hash -r 10000
function                    runs        average       minimum       maximum

hash:
sha1 (64 bytes)             10,000         241 ns         130 ns      14,740 ns
sha1 (4096 bytes)           10,000       2,845 ns       2,829 ns       9,504 ns
sha256 (64 bytes)           10,000         149 ns         145 ns         797 ns
sha256 (4096 bytes)         10,000       3,288 ns       3,278 ns       5,216 ns
sha512 (64 bytes)           10,000         317 ns         299 ns       4,164 ns
sha512 (4096 bytes)         10,000       7,314 ns       7,244 ns      14,505 ns
ripemd160 (64 bytes)        10,000         542 ns         532 ns       4,399 ns
ripemd160 (4096 bytes)      10,000      16,041 ns      15,981 ns      18,853 ns
sha3-256 (64 bytes)         10,000       2,495 ns       2,465 ns       4,130 ns
sha3-256 (4096 bytes)       10,000      91,921 ns      86,780 ns     129,577 ns
keccak256 (64 bytes)        10,000       2,490 ns       2,459 ns       4,246 ns
keccak256 (4096 bytes)      10,000      92,350 ns      86,832 ns     137,721 ns

…ives Create a generic micro-benchmarking framework intended for any functions. As the first application, alt_bn_128 adding/multiplying/pairing, sha3-256/keccak256 hashing, k1/r1/webauthn key signing/recovering, and modexp are benchmarked

larryk85 · 2022-08-17T22:01:09Z

benchmark/benchmark.cpp

+}
+
+void printt_header() {
+   std::cout << std::left << std::setw(name_width) << "function"


Is this supposed to be printt_header or print_header?

Thanks for your sharp eyes! Sorry for this copy/paste error.

spoonincode · 2022-08-17T22:08:07Z

benchmark/key.cpp

+   auto key = private_key::generate<ecc::private_key_shim>();
+
+   auto sign_f = [&]() {
+      key.sign(digest);


I would recommend disabling the canonical requirement for the signing. Otherwise the value could swing widely based on the private key you happened to randomly generate.

Good point! I will change to use fixed private keys so we can compare the values consistently across different runs.

if you just pass false here for the second parameter, it'll be constant time for all private keys anyways

Thanks! I am benchmarking both canonical and non-canonical cases.

imo there is very little utility in the canonical case -- I would never look at it personally. It's dependent on both the private key and payload, both of which are fixed here, thus making the time spent always be some constant multiplied with the non-canonical time. Up to you though.

I will remove the canonical case.

private key of k1 was randomly generated. It caused big swing in time with canonical requirement. This commit makes the private key fixed and benchmarks both canonical and non-canonical cases

spoonincode · 2022-08-21T02:08:14Z

In general LGTM but, in some conversation somewhere, didn't we think we wanted coverage of all hashes accessible via host functions? If anything, sha256 seems like a must have given its broad usage across the entire protocol.

linh2931 · 2022-08-21T14:44:37Z

I'll benchmark all crypto primitives which are exposed as host functions.

Make sure all exceptions are caught when executing code in http_plugin's thread pool, so Leap doesn't crash when processing APIs.

Transition to a temporary beta license

linh2931 requested review from heifner, larryk85, arhag and spoonincode August 17, 2022 21:56

larryk85 suggested changes Aug 17, 2022

View reviewed changes

spoonincode reviewed Aug 17, 2022

View reviewed changes

linh2931 added 2 commits August 17, 2022 18:10

benchmark: change printt_header to print_header

52bce93

benchmarking: make private key of k1 signing fixed

254d697

private key of k1 was randomly generated. It caused big swing in time with canonical requirement. This commit makes the private key fixed and benchmarks both canonical and non-canonical cases

linh2931 changed the title ~~[3.2] Create a benchmarking framework and use it to benchmark crypto primitives~~ [3.2] Create a simple benchmarking framework and use it to benchmark crypto primitives Aug 18, 2022

This was referenced Aug 18, 2022

[3.2] benchmarking framework and its use in benchmarking crypto primitives eosnetworkfoundation/mandel#799

Closed

FC::sha3.hpp: failed to compile on Ubuntu 18.04 with Boost 1.79 when included from benchmark/hash.cpp #31

Closed

Merge branch 'main' into benchmarking

88cfa65

k1 siging benchmarking: do not benchmark canonical case

cbe5cfb

linh2931 and others added 4 commits August 21, 2022 10:53

hash benchmarking: add sha1, sha256, sha512, and ripemd160

1fe04d3

benchmarking: add blake2 and remove unnecesary include header files

1e51b6c

blake2_benchmarking: correct the feature name from black2 to blake2

61e130b

Merge branch 'main' into benchmarking

096aa86

larryk85 approved these changes Aug 22, 2022

View reviewed changes

linh2931 merged commit 6b84267 into main Aug 22, 2022

linh2931 deleted the benchmarking branch August 22, 2022 19:32

greg7mdp added a commit that referenced this pull request Jul 26, 2023

Merge branch 'main' of github.com:AntelopeIO/leap into gh-17

513d510

greg7mdp added a commit that referenced this pull request Jul 27, 2023

Merge branch 'main' of github.com:AntelopeIO/leap into gh-17

2149f20

greg7mdp added a commit that referenced this pull request Jul 28, 2023

Merge pull request #1370 from AntelopeIO/gh-17

2971a61

Make sure all exceptions are caught when executing code in http_plugin's thread pool, so Leap doesn't crash when processing APIs.

heifner pushed a commit that referenced this pull request Apr 29, 2024

Merge pull request #17 from AntelopeIO/spring-beta-license

f937913

Transition to a temporary beta license

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[3.2] Create a simple benchmarking framework and use it to benchmark crypto primitives #17

[3.2] Create a simple benchmarking framework and use it to benchmark crypto primitives #17

linh2931 commented Aug 17, 2022 •

edited

Loading

larryk85 Aug 17, 2022

linh2931 Aug 17, 2022

spoonincode Aug 17, 2022

linh2931 Aug 17, 2022

spoonincode Aug 17, 2022

linh2931 Aug 18, 2022

spoonincode Aug 21, 2022

linh2931 Aug 21, 2022

spoonincode commented Aug 21, 2022

linh2931 commented Aug 21, 2022

[3.2] Create a simple benchmarking framework and use it to benchmark crypto primitives #17

[3.2] Create a simple benchmarking framework and use it to benchmark crypto primitives #17

Conversation

linh2931 commented Aug 17, 2022 • edited Loading

Help

Benchmarking all functions

Benchmarking a single feature with a different number of repeats from the default 1,000

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spoonincode commented Aug 21, 2022

linh2931 commented Aug 21, 2022

linh2931 commented Aug 17, 2022 •

edited

Loading